One solution is to use similar idea], which implements a [ across all of Unicode Stopwords These filters are described above under the relevant languages.įor other languages, the ASCIIFoldingFilterFactory won't do the foldings that you need. In this case it is better to not use ASCIIFoldingFilter before stemming, but instead to use the 'German2' stemmer first, which has language-specific handling for this case.įor some languages in non-Latin writing systems (Arabic, Greek, Hindi, Persian), there are filters to support the idea of 'diacritics-insensitive search' already included in Solr. But this might not be the best approach for your language, for example you may want ü to match to ue for German. If you are using a stemmer, most stemmers (especially Snowball stemmers) are a bit forgiving about diacritics, and these are handled on a language-specific basis.įor Latin-script writing systems, you can remove all diacritics with ASCIIFoldingFilterFactory. Some languages use diacritics, but people are not always consistent about typing them in queries or documents. It removes Greek diacritics and has special handling for the Greek final sigma. GreekLowerCaseFilterFactory: Use this instead of LowerCaseFilterFactory for the Greek language.It includes special handling for dotted and dotless I. TurkishLowerCaseFilterFactory: Use this instead of LowerCaseFilterFactory for the Turkish language.

However, some languages have special casing properties, and these have their own filters: In most cases LowerCaseFilterFactory is sufficient. In general most languages don't require special tokenization (and will work just fine with Whitespace + WordDelimiterFilter), so you can safely tailor the English 'text' example schema definition to fit. If you have problems (your language is highly-inflectional, etc), you might want to try using an n-gram approach as an alternative. Remember, things like stemming and stopwords aren't mandatory for the search to work, only optional language-specific improvements. A first step is to start with the 'textgen' type in the example schema. Note: Be sure to use the Turkish-specific TurkishLowerCaseFilterFactory Solr3.1 My language is not listed!!! These two are simple rule based stemmers, not handing exceptions or irregular forms.Įxample set of Turkish stopwords (Be sure to switch your browser encoding to UTF-8) Since Solr3.6 you can also use solr.NorwegianLightStemFilterFactory for a lighter variant or solr.NorwegianMinimalStemFilterFactory attempting to normalize plural endings only. Solr includes support for stemming Norwegian via solr.SnowballPorterFilterFactory, and Lucene includes an example stopword list. Note: Be sure to use PositionFilter at query-time (only) as these languages do not use spaces between words. To use this tokenizer, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to add to your SOLR_HOME/lib Lucene provides support for segmenting these languages into syllables with solr.ICUTokenizerFactory in the analysis-extras contrib module.

Example set of Italian stopwords (Be sure to switch your browser encoding to UTF-8) Lao, Myanmar, Khmer

Apache lucene relevance models