American physicians are increasingly unhappy with their once-vaunted profession, and that malaise is bad for their patients.

All too often these days, I find myself fidgeting by the doorway to my exam room, trying to conclude an office visit with one of my patients. When I look at my career at midlife, I realize that in many ways I have become the kind of doctor I never thought I’d be: impatient, occasionally indifferent, at times dismissive or paternalistic. Many of my colleagues are similarly struggling with the loss of their professional ideals.

It could be just a midlife crisis, but it occurs to me that my profession is in a sort of midlife crisis of its own. In the past four decades, American doctors have lost the status they used to enjoy. In the mid-20th century, physicians were the pillars of any community. If you were smart and sincere and ambitious, at the top of your class, there was nothing nobler or more rewarding that you could aspire to become.

Today medicine is just another profession, and doctors have become like everybody else: insecure, discontented and anxious about the future. In surveys, a majority of doctors express diminished enthusiasm for medicine and say they would discourage a friend or family member from entering the profession. In a 2008 survey of 12,000 physicians, only 6% described their morale as positive. Eighty-four percent said that their incomes were constant or decreasing. Most said they didn’t have enough time to spend with patients because of paperwork, and nearly half said they planned to reduce the number of patients they would see in the next three years or stop practicing altogether.

American doctors are suffering from a collective malaise. We strove, made sacrifices—and for what? For many of us, the job has become only that—a job.

That attitude isn’t just a problem for doctors. It hurts patients too.

In a survey of 12,000 physicians, only 6% described their morale as positive.

Consider what one doctor had to say on Sermo, the online community of more than 270,000 physicians:

“I wouldn’t do it again, and it has nothing to do with the money. I get too little respect from patients, physician colleagues, and administrators, despite good clinical judgment, hard work, and compassion for my patients. Working up patients in the ER these days involves shotguning multiple unnecessary tests (everybody gets a CT!) despite the fact that we know they don’t need them, and being aware of the wastefulness of it all really sucks the love out of what you do. I feel like a pawn in a moneymaking game for hospital administrators. There are so many other ways I could have made my living and been more fulfilled. The sad part is we chose medicine because we thought it was worthwhile and noble, but from what I have seen in my short career, it is a charade.”

The discontent is alarming, but how did we get to this point? To some degree, doctors themselves are at fault.

In the halcyon days of the mid-20th century, American medicine was also in a golden age. Life expectancy increased sharply (from 65 years in 1940 to 71 years in 1970), aided by such triumphs of medical science as polio vaccination and heart-lung bypass. Doctors largely set their own hours and determined their own fees. Popular depictions of physicians (“Marcus Welby,” “General Hospital”) were overwhelmingly positive, almost heroic.

American doctors at midcentury were generally content with their circumstances. They were prospering under the private fee-for-service model, in which patients were covering costs out of pocket or through fledgling private insurance programs such as Blue Cross/Blue Shield. They could regulate fees based on a patient’s ability to pay and look like benefactors. They weren’t subordinated to bureaucratic hierarchy.

Read more from The Wall Street Journal: Five Things To Know Today.

After Medicare was introduced in 1965 as a social safety net for the elderly, doctors’ salaries actually increased as more people sought medical care. In 1940, in inflation-adjusted 2010 dollars, the mean income for U.S. physicians was about $50,000. By 1970, it was close to $250,000—nearly six times the median household income.

But as doctors profited, they were increasingly perceived as bilking the system. Year after year, health-care spending grew faster than the U.S. economy as a whole. Meanwhile, reports of waste and fraud were rampant. A congressional investigation found that in 1974, surgeons performed 2.4 million unnecessary operations, costing nearly $4 billion and resulting in nearly 12,000 deaths. In 1969, the president of the New Haven County Medical Society warned his colleagues “to quit strangling the goose that can lay those golden eggs.”

If doctors were mismanaging their patients’ care, someone else would have to manage that care for them. Beginning in 1970, health maintenance organizations, or HMOs, were championed to promote a new kind of health-care delivery built around price controls and fixed payments. Unlike with Medicare or private insurance, doctors themselves would be held responsible for excess spending. Other novel mechanisms were introduced to curtail health outlays, including greater cost-sharing by patients and insurer reviews of the necessity of medical services. That ushered in the era of HMOs.

Public opinion of doctors shifted distinctly downward too. Doctors were no longer unquestioningly exalted. On television, physicians were portrayed as more human—flawed or vulnerable (“M*A*S*H*,” “St. Elsewhere”) or professionally and personally fallible (“ER”).

As managed care grew (by the early 2000s, 95% of insured workers were in some sort of managed-care plan), physicians’ confidence plummeted. In 2001, 58% of about 2,000 physicians questioned said that their enthusiasm for medicine had gone down in the previous five years, and 87% said that their overall morale had declined during that time. More recent surveys have shown that 30% to 40% of practicing physicians wouldn’t choose to enter the medical profession if they were deciding on a career again—and an even higher percentage wouldn’t encourage their children to pursue a medical career.

There are many reasons for this disillusionment besides managed care. One unintended consequence of progress is that physicians increasingly say they don’t have enough time to spend with patients. Medical advances have transformed once-terminal diseases—cancer, AIDS, congestive heart failure—into complex chronic conditions that must be managed over the long term. Physicians also have more diagnostic and treatment options and must provide a growing array of screenings and other preventative services.

At the same time, salaries haven’t kept pace with doctors’ expectations. In 1970, the average inflation-adjusted income of general practitioners was $185,000. In 2010, it was $161,000, despite a near doubling of the number of patients that doctors see a day.

While patients today are undoubtedly paying more for medical care, less of that money is actually going to the people who provide the care. According to a 2002 article in the journal Academic Medicine, the return on educational investment for primary-care physicians, adjusted for differences in number of hours worked, is just under $6 per hour, as compared with $11 for lawyers. Some doctors are limiting their practices to patients who can pay out of pocket without insurance company discounting.

Other factors in our profession’s woes include a labyrinthine payer bureaucracy. U.S. doctors spend almost an hour on average each day, and $83,000 a year—four times their Canadian counterparts—dealing with the paperwork of insurance companies. Their office staffs spend more than seven hours a day. And don’t forget the fear of lawsuits; runaway malpractice-liability premiums; and finally the loss of professional autonomy that has led many physicians to view themselves as pawns in a battle between insurers and the government.

The growing discontent has serious consequences for patients. One is a looming shortage of doctors, especially in primary care, which has the lowest reimbursement of all the medical specialties and probably has the most dissatisfied practitioners. Try getting a timely appointment with your family doctor; in some parts of the country, it is next to impossible. Aging baby boomers are starting to require more care just as aging baby boomer physicians are getting ready to retire. The country is going to need new doctors, especially geriatricians and other primary care physicians, to care for these patients. But interest in primary care is at an all-time low.

Perhaps the most serious downside, however, is that unhappy doctors make for unhappy patients. Patients today are increasingly disenchanted with a medical system that is often indifferent to their needs. People used to talk about “my doctor.” Now, in a given year, Medicare patients see on average two different primary care physicians and five specialists working in four separate practices. For many of us, it is rare to find a primary physician who can remember us from visit to visit, let alone come to know us in depth or with any meaning or relevancy.

Insensitivity in patient-doctor interactions has become almost normal. I once took care of a patient who developed kidney failure after receiving contrast dye for a CT scan. On rounds, he recalled for me a conversation he’d had with his nephrologist about whether his kidney function was going to get better. “The doctor said, ‘What do you mean?’ ” my patient told me. “I said, ‘Are my kidneys going to come back?’ He said, ‘How long have you been on dialysis?’ I said, ‘A few days.’ And then he thought for a moment and said, ‘Nah, I don’t think they’re going to come back.’ ”

My patient broke into sobs. ” ‘Nah, I don’t think they’re going to come back.’ That’s what he said to me. Just like that.”

Of course, doctors aren’t the only professionals who are unhappy today. Many professions, including law and teaching, have become constrained by corporate structures, resulting in loss of autonomy, status, and respect. But as the Princeton sociologist Paul Starr writes, for most of the 20th century, medicine was “the heroic exception that sustained the waning tradition of independent professionalism.” It is an exception whose time has expired.

How can we reverse the disillusionment that is so widespread in the medical profession? There are many measures of success in medicine: income, of course, but also creating attachments with patients, making a difference in their lives and providing good care while responsibly managing limited resources.

The challenge in dealing with physician burnout on a practical level is to create new incentive schemes to foster that meaning: publicizing clinical excellence, for example (public reporting of surgeons’ mortality rates or physicians’ readmission rates is a good first step), or giving rewards for patient satisfaction (physicians at my hospital now receive quarterly reports that tell us how our patients rate us on measures such as communication skills and the amount of time we spend with them).

We also need to replace the current fee-for-service system with payment methods such as bundled payment, in which doctors on a case are paid a lump sum to divide among themselves, or pay for performance, which offers incentives for good health outcomes. We need systems that don’t simply reward high-volume care but also help restore the humanism in doctor-patient relationships that have been weakened by business considerations, corporate directives and third-party intrusions.

I believe most doctors continue to want to be like the physician knights of the golden age of medicine. Most of us went into medicine to help people. We want to practice medicine the right way, but too many forces today are propelling us away from the bench or the bedside. No one ever goes into medicine to do unnecessary testing, but this sort of behavior is rampant. The American system too often seems to promote knavery over knighthood.

Fulfillment in medicine, as with any endeavor, is about managing hopes. Probably the group best equipped to deal with the changes wracking the profession today is medical students, who are not so weighed down by great expectations. Doctors ensconced in professional midlife are having the hardest time.

In the end, the problem is one of resilience. American doctors need an internal compass to navigate the changing landscape of our profession. For most doctors, this compass begins and ends with their patients. In surveys, most physicians—even the dissatisfied ones—say the best part of their jobs is taking care of people. I believe this is the key to coping with the stresses of contemporary medicine: identifying what is important to you, what you believe in and what you will fight for. Medical schools and residency programs can help by instilling professionalism early on and assessing it frequently throughout the many years of training. Introducing students to virtuous mentors and alternative career options, such as part-time work, may also help stem some of the burnout.

What’s most important to me as a doctor, I’ve learned, are the human moments. Medicine is about taking care of people in their most vulnerable states and making yourself somewhat vulnerable in the process. Those human moments are what others—the lawyers, the bankers—envy about our profession, and no company, no agency, no entity can take those away. Ultimately, this is the best hope for our professional salvation.