4D CT imaging has great potential for use in stroke workup. A fully convolutional neural network (CNN) for 3D multiclass segmentation in 4D CT is presented, which can be trained end-to-end from sparse 2D annotations. The CNN was trained and validated on 42 4D CT acquisitions of the brain of patients with suspicion of acute ischemic stroke. White matter, gray matter, cerebrospinal fluid, and vessels were annotated by two trained observers. The mean Dice coefficients, contour mean distances, and absolute volume differences were respectively 0.87+-0.04, 0.52+-0.47 mm, and 11.78+-9.55 \% on a separate test set of five patients, which were similar to the average interobserver variability scores of 0.88+-0.03, 0.72+-0.93 mm, and 8.86+-7.65 \% outperforming the current state-of-the-art. The proposed method is therefore a promising deep neural network for multiclass segmentation in 4D spatiotemporal imaging data.