It is the burden of difficult and hard to follow religious laws.
The theme of the second sura is the book (in the sense of religious laws from God)
and its effect as a guide on people.
You can see this in the introduction section of the sura
on the first and second pages of the sura.
According to those verses
there are three groups of people regarding the effect of the book:
those who are mindful of God (الْمُتَّقِينَ),
those who disbelieve (الَّذِينَ كَفَرُوا), and
those whom there is a sickness in their heart (الَّذِينَ فِي قُلُوبِهِم مَّرَضٌ).
The third group makes up the middle section between the previous two groups.
The first group the book guides surely (verses 2:2-5),
the second group it won't guide them no matter what (verses 2:6-7).
The effect of the book on the third group is more complicated and
a major part of the sura is devoted to discussing them.
You can read some descriptions of this group on the second page (verse 2:8-20).
Those who follow the book will join the mindful,
those who don't will join the disbelievers.
This group seem to be people who claim to be believers
(and it seems they do think they are)
but they are not really believers in heart.
So they are religious people who are not real believers.
Almost half of the sura is dedicated to discussing them,
until the story of Ibrahim (PBUH).
These people are common in religious societies.
Muslims were not the first religious society.
The people of the book preceded them.
Quran uses the previous religious societies and
their behavior as an example to discuss this group,
so Muslims may learn and avoid those mistakes.
The sura mostly focuses on the followers of Judaism.
The final section of the sura should be understood with these in mind.
There was a religious community before Muslims,
i.e. the followers of Judaism.
Now there is a new religion and the description of its laws
constitute the main part of the second half of the sura.
So the people who were before Muslims are the people of the book,
mainly the followers of the Judaism
(and this is also consistent with usage of term in other places).
So the question is what was the burden (إِصْرًا) that was put on their back?
It seems to be the burden of difficult religious laws
(and verse 62:5 adds support to this theory).
If you look at their behavior in the sura here is what typically happens:
they are ordered to behave in some way
(e.g. kill a cow, verse 2:67),
they don't perform it as required because they are not believers in heart,
so the order becomes more strict and more difficult to perform
(e.g. kill the best cow you have,
with very detailed description of the best cow, verses 2:68-71).
If you read about the laws of Judaism (Halakha)
it is way more difficult than the laws of Islam (Sharia).
This is confirmed also by narrations from the Prophet.
You may ask why laws become more difficult?
The reason is that
the religious people who are not believers in heart
try to circumvent the laws and transgress them,
so the laws become stricter to accommodate them and
make them harder to circumvent.
The prayer at the end of the sura matches the verses about the mindful in the introduction
That part of the prayer
is Muslims asking God
not to make the laws of Islam as difficult as he made the laws of Judaism,
but forgive when Muslims transgress by mistake or forgetfulness.
As a general rule,
when a prater is stated in Quran
it means that God has accepted that prater
(unless it is said otherwise in Quran).
We can see that Islam is way easier than Judaism to practice.
That is not because Muslims did not transgress the laws, but
because God often forgave and reduced the burden and made it easier to follow
(for an example see verse 2:187 where it says God passed on from you (عَفَا عَنكُمْ)).
(This doesn't mean that the laws didn't become difficult and
will not become difficult in future,
e.g. when people really know what is expected from them
but ask for fatwas seeking a way to circumvent them,
the Prophet and after him the scholars
might make the laws more stricter.
Muslims are advised not ask the kind of questions that
those preceding them asked from their prophets,
see verse 2:108.)
This is what I understand, and surely God knows best.