Colon cancer is a kind of cancer that originates in the large intestine (colon) and is highly lethal accounting as the third dominant reason for deaths related to cancer globally. Cancer detection at a premature stage substantially increases the probability of survival. With the increased use of image processing tools in medical imaging, several techniques have been developed to detect colon cancer at a considerably less cost and time consumption. In this paper a highly efficient hybrid optimization technique is introduced using Convolutional Neural Networks (CNN) with transfer learning for colon cancer detection. CNN with transfer learning is utilized to analyse the segmented images of the colon and classifying them as cancerous or non-cancerous. The CNN is used with the hyper parameters from trained models, like Alexnet. A new Remora Shuffled Shepherd Optimization Algorithm (RSSOA) algorithm was introduced by incorporating the Shuffled Shepherd Optimization Algorithm (SSOA) and Remora Optimization Algorithm (ROA) for updating the weights of the hidden neurons in the CNN. The devised approach is accessed for its performance by considering different metrics, like accuracy sensitivity, and specificity, Confusion matrix.