Transformer‑Based Denoising AutoEncoder (TSDAE) for Job Description Embeddings (Job2Vec)

TSDAE is a domain‑adaptive pre‑training method for sentence embeddings that outperforms approaches such as Masked Language Models.

It trains by adding noise (e.g., deleting or swapping words) to input sentences, encoding the corrupted sentence into a fixed‑size vector, and decoding it back to the original text; the decoder only receives the encoder’s sentence representation, creating a bottleneck that forces meaningful embeddings.

TSDAE’s S‑BERT implementation adds noise to the input text and removes roughly 60% of the words. The encoder maps this noisy input to a fixed‑size sentence embedding, and the decoder attempts to reconstruct the clean sentence. After training, only the encoder is used to generate embeddings.

The training loss, DenoisingAutoEncoderLoss, expects batches composed of pairs [noise_fn(sentence), sentence]. Batches are built with DenoisingAutoEncoderDataset, where sentence is a list of strings and noise_fn returns a noisy version of a given string.

param sentence: a list of sentences

param noise_fn: a function that takes a string and returns a noisy string (e.g., with deleted words)

After cleaning a job‑requirements dataset, the document length distribution is visualized, and the cleaned sentence list is fed directly into the model.

Training for five epochs produces embeddings for sample data, shown in the accompanying figures.

FAISS is then used to index these embeddings, enabling similarity search to retrieve matching job descriptions; the retrieval results appear promising.