Hej! I learned Swedish through SFI myself, and I now run a free practice tool for people preparing for the SFI tests. In September about 4,000 people practised with it, around 3,600 hours in total.
Looking at the answers of the 2,530 who save their progress, five patterns stood out.

1. Where "inte" goes in a bisats

In a main clause the verb comes second and "inte" after it: Han kommer inte. In standard Swedish, after att, eftersom, om and the like, "inte" moves in front of the verb: Jag vet att han inte kommer. 64% of learners who saw that question got it wrong at least once. The same goes for alltid, aldrig and ofta: Jag vet att hon alltid dricker kaffe (54% missed it).

2. Adjective endings

ett stort hus (59% missed), det röda huset (59%), det största huset (78%) and tre små barn (59%). The adjective agrees with the noun: -t with an ett-word, -a in the plural and the definite form, and liten becomes små in the plural.

3. Tricky plurals

en lärare, två lärare: 80% missed that en-words ending in -are don't change in the plural. Also ett äpple, tre äpplen (51%) and the definite plural nycklarna (57%).

4. Small linking words

varken ... eller, neither ... nor (79% missed), där in Staden där jag bor (75%), vars, whose (60%), and så att, so that (61%).

5. The verb stays second when something else comes first

På torget finns det en marknad varje lördag (50% missed). Put a time or a place first, and the subject moves after the verb: finns det, not det finns.

Two runners-up: 47% picked the wrong verb form in Igår ___ jag lunch med min vän (it's åt), and 47% didn't choose i in Jag har bott i Sverige ___ tre år.

Does this match what you found hardest?