You can use the $- special parameter to know if the shell is interactive. Combine that with substitution in the parameter expansion and you can isolate just the 'i' that indicates it's an interactive shell. Assign the result to a environment variable that will either be the 'i' or empty, and that can be used with the [[, [, or test builtin commands that treat an empty string as false to determine whether or not to do something.
INTERACTIVE_SHELL=${-//[^i]}
if [ $INTERACTIVE_SHELL ]; then
source ~/my_interactive_stuff.sh
fi