Under the right care and conditions, your plant will grow happy and healthy. But here are some issues you may encounter while caring for an Olive Tree: 

Brown spots, holes, nibbled edges, insects and mold - When spotted, remove infected leaves immediately and apply neem oil.

Yellowing leaves - This is due to nutrient deficiency. Add organic matter to soil or fast uptake speed balanced fertiliser. 

Shedding leaves - Usually caused by lack of sunlight. Move your Olive Tree to a windowsill or somewhere with direct sunlight. 

Brown leaf edges - This is caused by underwatering. Increase frequency of watering and allow soil to soak thoroughly through soil. 

Droopy but not dry leaves - The plant might be overwatered. Water deeply but less frequently and ensure proper drainage.