AutoJev-27B ranks second on Hugging face on Jev benchmarks. Here is a quick summary of architecture -
AutoJev-27B is a pretrained Qwen3.8-27B backbone with its generative vocabulary (LM head) readout replaced by a small bounded decision head.
The decision head is a linear projection from the 5,120-dimensional final hidden state to 255 option logits.
A logit is a raw score the model gives each option, and a softmax turns those scores into probabilities that add up to 1.
a. Each of the 255 outputs maps to an option label such as A, B or C that the Qwen tokenizer encodes as a single token. If a question has three options, only those three logits are used and the rest are masked.
b. The decision head is not randomly initialized. AutoJev copies the rows for the option label tokens from Qwen's original output layer. The model starts with weights that already relate to tokens such as A, B and C, and then the full model is fine-tuned.
c. The context, the question and all options go into one input sequence. Qwen processes them together. AutoJev takes the hidden state at the last position and passes it to the decision head.
d. A softmax over the active option logits, scaled by a calibrated temperature, gives the probability of each option. There is no decoding loop and no text is generated.
e. The same setup handles yes/no questions.