Tensions are brewing at work between employees who have returned to the office and those who’ve continued to work from home, according to a survey of U.S. workers.

Almost three-quarters of respondents said companies should pay in-office employees more than their work-from-home colleagues, and two-thirds are concerned that managers view full-time remote workers as lazier, according to the survey of 3,500 people commissioned by GoodHire, a firm that performs employment background checks.

At the same time, a third of respondents are willing to quit their job or start looking for a new one if forced to return to the office full-time. That share is down from last year, underscoring growing pressures in workplaces. A majority think that work-from-home employees will be more at risk to lose their jobs in a downturn.

Both camps agree on one important matter: Working remotely will probably hamper their career.

“There are several data points in the survey that show there may be a potential for a growing conflict among these two groups,” Max Wesman, chief operating officer at GoodHire, said in an emailed statement. Still, “each supports the general idea that in-office workers will enjoy more benefits and career opportunities than their remote-worker counterparts.”

Many corporations, especially banks, have taken a harder line on getting employees back to the office at least part-time. But data show there’s still a long way to go, particularly in big cities where many are still working remotely at least a few days during the week.

The GoodHire poll shows that a majority of workers feel that, while they might prefer remote work, they believe that their careers will suffer. Almost 6 in 10 worried that they would be excluded from important team meetings and projects if they’re not consistently in the office.