Brygmophyseter is a member of a fossil stem group of hyper-predatory macroraptorial sperm whales from the Miocene (often shortened to "raptorial") The other members are Acrophyseter, Livyatan, and Zygophyseter, and these four whales have in common enamel-coated teeth in both the upper and lower jaws which were used in hunting large prey. These teeth are thought to have evolved in either a basilosaurid-like common ancestor, or multiple times independently in the group. Brygmophyseter is the oldest raptorial. It has been proposed that these raptorials be placed into the extinct, paraphyletic subfamily Hoplocetinae, alongside Scaldicetus, Diaphorocetus, Idiorophus, and Hoplocetus. That is to say that the subfamily does not consist a common ancestor and all of its descendants.
The species was first described in 1994 by paleontologists Kiyoharu Hirota and Lawrence Barnes. Observing the teeth which were present in both jaws and coated in enamel, they had placed it into the genus Scaldicetus which was described in 1867 as having similar characteristics. However, a 1995 study stated that since Scaldicetus is known only from tooth remains, new species should not be assigned to the genus. Therefore, the study authored by paleontologists Kimura Toshiyuki, Hasegawa Yoshikazu, and Barnes Lawrence placed it into the newly erected genus Brygmophyseter. Scaldicetus is now used as a wastebasket taxon. A 2006 study by geologists Giovanni Bianucci and Walter Landini placed the whale into the genus Naganocetus which they had created, and concluded that it was most closely related to Zygophyseter, which they had for the first time described in the same study. This was justified by their similar body sizes, the basin containing the spermaceti organ (the supracranial basin) not extending forward (causing a convex snout), and the presence of long temporal fossae on the sides of the skull. However, since Brygmophyseter was published first, Naganocetus is considered to be a junior synonym.
The holotype specimen, SFM-0001, was excavated from the Bessho Formation in the Nagano Prefecture in Japan in 1988 by the residents of Shiga-mura with assistance from the staff of the Shiga Fossil Museum. The specimen is nearly complete, and consists of a partial skull, the jawbone, the lingual bone in the neck, vertebrae, ribs, the breastbone, and humeri and radii in the limbs. It was dated to the Langhian stage of the Miocene 14–15 million years ago (mya), and the specimen is currently on display at the Gunma Museum of Natural History in Japan.
The genus name is a combination of the Greek word brygmos, which means "biting" or "gnashing", combined with physeter, which is the generic name of the living sperm whale, and which is also the Greek word for "blower". In the junior synonym Naganocetus derives from the prefecture it was found in, the Nagano Prefecture, and the Latin cetus meaning "whale".
Like other raptorials, Brygmophyseter had enamel-coated teeth in both jaws, unlike the modern sperm whale (Physeter macrocephalus). It probably had 11 or 12 teeth in each jaw, though no upper teeth were preserved in the holotype. The skull of the holotype measured around 1.5 meters (4.9 ft), and it had an elongated snout. The brow ridge was broad and flat-topped, and the zygomatic process of the squamosal bone on the cheeks was large and robust. The lower jaw probably contained 12 teeth, and the upper jaw probably had a similar tooth count. The temporal fossa on the sides of the skull were elongated, which may have been plesiomorphic features from archaeocetes, that is, it is an ancestral characteristic of the whale. The right nasal passage was small and was asymmetrical with the left nasal passage, like in the modern sperm whale. Characteristic of sperm whales, it had a deep basin on the top of its skull, known as the supracranial basin. Like in the modern sperm whale, this basin probably held the spermaceti organ, and so the whale had biosonar capabilities. The nuchal crest which is the part of the whale skull which projects upwards on the back end of the skull, behind the supracranial basin, was low and broad. Similar to modern day toothed whales, but unlike in the modern sperm whale, the shoulder blades were thicker than they were taller.
Brygmophyseter is estimated to have been around 7 meters (23 ft) long. The holotype had preserved zero neck vertebrae, ten thoracic vertebrae, ten lumbar vertebrae, and fifteen tail vertebrae. In comparison, the modern sperm whales has eleven thoracic, eight lumbar, and twenty two tail vertebrae, and the smaller tail in Brygmophyseter is probably a primitive characteristic. The vertebra segments increased in height until the seventh lumbar vertebra, then they began to decrease. In the thoracic vertebrae, with the exception of the tenth one, the width of the segments is larger than the height. The lumbar and tail vertebrae are circular in shape.
The head of the humerus arm bone of Brygmophyseter was positioned perpendicular to the shaft. The elbow joint on the humerus was not distinct from the ulna, typical in cetaceans. The ulna was more primitive than that of the modern sperm whale, in that the shaft was longer and more slender, and the head of the bone was smaller.
Since Brygmophyseter was a raptorial, and the group is characterized by their adaptations for subduing large prey, Brygmophyseter was likely a macropredator of marine mammals and other large marine vertebrates, occupying a niche similar to the killer whale. Though no stomach remains or bite marks have been found, it is thought to have preyed upon a variety of animals, including whales, seals, fish, and cephalopods.
Brygmophyseter was discovered in the Middle Miocene deposits of the Bessho Formation of Japan, which has also yielded some of the earliest oceanic dolphins and rorqual baleen whales, as well as beaked whale remains. The formation also had fossil bivalves which indicate the presence of deep sea hydrothermal vents, and there is also evidence of prehistoric cold vents in the area, which today host chemosynthetic specialist species; the formation was likely 1,000–2,000 meters (3,300–6,600 ft) deep in the Miocene. Also found were fossil plants, echinoderms, cephalopods, deep-sea teleost fish, and seabirds. The shark remains discovered in this formation belong to the sandbar shark (Carcharhinus plumbeus) and other unidentified Carcharhinus sharks, the seal shark (Dalatias licha), the extinct broad-toothed mako (Isurus hastalis), the extinct hook-tooth mako (Isurus planus), the Japanese velvet dogfish (Scymnodon ichiharai), and a species of Etmopterus lantern shark.
Brygmophyseter was featured in the fifth episode of The History Channel's Jurassic Fight Club, "Deep Sea Killers". In this episode, Brygmophyseter, which was referred to as the "biting sperm whale," was portrayed as being able to weaponize sonar in order to stun prey, and to have traveled in pods like the modern day killer whale. The pod was attacked by a megalodon, which seemingly had a similar size to the whales. When the megalodon severely wounded one of the members of the pod, the rest of the whales teamed up to drive the megalodon off. Once the member of the pod died, however, the megalodon returned to feed on it.