Purpose

app.js handles 3 things. It handles HTTP requests for compiles, it serves the resulting PDFs, it also does some operational tasks such as reporting the load to WebSocketloadbalancer.js so that service don’t get overloaded.

Details & Features

  1. Web Server Timeouts.
const TIMEOUT = 10 * 60 * 1000 //10 Minutes Timeout cuz compilation is slow
app.use(function (req, res, next) {
	req.setTimeout(TIMEOUT)
	res.setTimeout(TIMEOUT) 
	res.removeHeader('X-Powered-By')
	next()
})
  1. Security - Parameter Validation
app.param('project_id', function (req, res, next, projectId) {
  if (projectId?.match(/^[a-zA-Z0-9_-]+$/)) {
    next()
  } else {
    next(new Error('invalid project id'))
  }
})
  1. Core Logic
// The Main Event: Compile
app.post(
  '/project/:project_id/compile',
  bodyParser.json({ limit: Settings.compileSizeLimit }), // Limit payload size
  CompileController.compile // -> Goes to CompileController.js
)

// SyncTeX: Clicking PDF jumps to Code
app.get('/project/:project_id/sync/code', CompileController.syncFromCode)
app.get('/project/:project_id/sync/pdf', CompileController.syncFromPdf)

// Word Count
app.get('/project/:project_id/wordcount', CompileController.wordcount)

Let’s take the main event, compile, as an example. app.post is essentially saying: when someone sends a request to /project/:project_id/compile (basically happens when you click the compile ui), it’s going to perform the following operations: bodyParser.json({limit: Settings.compileSizeLimit}), and CompileController.compile. the first line is just a simple payload limiter, you probably have to pay for larger payload to adjust the Settings.compileSizeLimit if you want to edit larger file, the second line is the core, we can see that this operation is sending it to CompileController , so now it’d be nice for us to start investigating into CompileController.js (Link at bottom of the page)

Of course when user clicks the Compile button it’s not just one thing happening, we also have to perform some other things to make a comprehensive, well-thought editor. For example, after the user clicks on Compile, the button UI will change to “stop”, and user can click that, and if user clicks on stop, that’s another POST request to the CompileController to stop the compilation through .stopCompile as you can see in the first line of code below.

app.post('/project/:project_id/compile/stop', CompileController.stopCompile)
app.delete('/project/:project_id', CompileController.clearCache)
app.get('/project/:project_id/sync/code', CompileController.syncFromCode)
app.get('/project/:project_id/sync/pdf', CompileController.syncFromPdf)
app.get('/project/:project_id/wordcount', CompileController.wordcount)
app.get('/project/:project_id/status', CompileController.status)
app.post('/project/:project_id/status', CompileController.status)
  1. Serving the PDF

const ForbidSymlinks = require('./app/js/StaticServerForbidSymlinks')

// create a static server which does not allow access to any symlinks
// avoids possible mismatch of root directory between middleware check
// and serving the files
const staticOutputServer = ForbidSymlinks(
  express.static,
  Settings.path.outputDir,
  {
    setHeaders(res, path, stat) {
      if (Path.basename(path) === 'output.pdf') {
        // Calculate an etag in the same way as nginx
        // <https://github.com/tj/send/issues/65>
        const etag = (path, stat) =>
          `"${Math.ceil(+stat.mtime / 1000).toString(16)}` +
          '-' +
          Number(stat.size).toString(16) +
          '"'
        res.set('Etag', etag(path, stat))
      }
      res.set('Content-Type', ContentTypeMapper.map(path))
    },
  }

  1. ForbidSymLinks is another security measure: if a user uploads .Tex file that generates a symlink to /root/.ssh, and then requests that symlink via the web server, then you are cooked bro. they could steal secrets. This wrapper explicitly blocks that. You’d have to read StaticServerForbidSymlinks.js for more details.

  2. TCP Load Balancer

//Lines 278–335
const loadTcpServer = net.createServer(function (socket) {
  // ...
  const currentLoad = os.loadavg()[0]
  const freeLoad = availableWorkingCpus - currentLoad
  
  socket.write(`up, ready, ${Math.max(freeLoadPercentage, 1)}%\\n`, 'ASCII')
  // ...
})

This is a TCP load balancer, it’s different from the WebSocketLoadBalancer previously configured in the file. The Websocket Load Balancer manages people (connections). This TCP Agent manages work (CPU cycles). Because compilation is so heavy, we need this specialized "smart" agent to avoid crashing the servers with too much simultaneous work. compilation itself is using the CPU, so websocket load balancer won’t know this, it just knows how many people are connecting to where. For TCP load balancer we are able to distribute load for for heavy compilations.