美文网首页
Lucene和Solr服务封装

Lucene和Solr服务封装

作者: 艾剪疏 | 来源:发表于2018-08-26 02:00 被阅读50次

1 架构设计
2 工厂模式启动服务
3 Lucene服务
4 Solr服务

这个是将Lucene和Solr的服务进行封装成独立的工具类,提供外部调用的方法,后可以打jar将其导入到其他的项目中直接使用,是一种很好的思想。

1 架构设计

根据功能,项目的架构设计如图。


image.png

各个部分在图中已有说明,下面将主要功能代码予以说明。
1 索引参数

public class FullTextIndexParams {
    //需要索引的数据
    private List<Map<String,Object>> indexData = new ArrayList<Map<String,Object>>();
    //lucene 的索引路径
    private String indexPath = "";
    //删除索引时,传入id
    private List<String> ids = new ArrayList<String>();
    private String id;
}

2 搜索结果

public interface FullTextResult {
    //返回搜索集合
    public List getResultList();    
    public void setResultList(List list);   
    //返回统计搜索集合
    public List getFacetList(); 
    public void setFacetList(List list);    
    //返回总记录数
    public long getNumFound();  
    public void setNumFound(long numFound);
}

3 搜索参数

public class FullTextSearchParams {

    //搜索关键词
    private String queryWord = "";

    //指定搜索域,并且默认的关系是OR
    private List<String> assignmentFields = new ArrayList<String>();
    /**
     * 指定搜索域与搜索域之间的关系
     * Map<String,String> 第一个String是域名,比如:name
     *                    第二个String是关系,写法:或者:OR 与:AND
     */
    private List<Map<String,String>> assignFields = new ArrayList<Map<String,String>>();

    //显示域
    private String[] viewFields;

    //是否高亮
    private Boolean isHighlight = true;

    //高亮域
    private String[] highlightFields;

    //高亮前缀
    private String preHighlight = "<em class=\"highlight\">";

    //高亮后缀
    private String postHighlight = "</em>";

    //排序域 String:需要排序的域名,Boolean:true 升序 false 降序
    private Map<String,Boolean> sortField = new HashMap<String,Boolean>();

    //过滤域
    private Map<String,String> filterField = new HashMap<String,String>();

    //开始行
    private int startNums = 0;

    //一页显示多少行
    private int pageCount = 15;

    //是否统计
    private Boolean isFacet = false;

    //统计域
    private String[] facetFields;

    //返回的结果数
    private long numFound;

    //显示结果的字数
    private int viewNums = 200;

    //指定搜索权重
    private Map<String,Float> boost = new HashMap<String,Float>();

    //指定lucene中返回的结果数
    private int returnNums = 1000;
}

4 所有服务接口

public interface FullTextService {

    //启动服务
    public int beginService(String serverName);

    //启动服务,带启动地址
    public int beginService(String serverName,String url);

    //flag: 0:IndexWriter 1:IndexSearcher
    public int beginService(String serverName,String flag,String indexPath);

    //设置服务name
    public void setServerName(String serverName);

    //关闭服务
    public int endService(String serverName);

    //索引
    public void doIndex(FullTextIndexParams fullTextIndexParams);

    //索引之前要做的事情
    public void preIndexMethod();

    //索引之后要做的事情
    public void afterIndexMethod();

    //修改索引
    public void updateIndex(FullTextIndexParams fullTextIndexParams);

    //修改之前要做的事情
    public void preUpdateIndexMethod();

    //修改之后要做的事情
    public void afterUpdateIndexMethod();

    //删除索引
    public void deleteIndex(FullTextIndexParams fullTextIndexParams);

    //删除之前要做的事情
    public void preDeleteIndexMethod();

    //删除之后要做的事情
    public void afterDeleteIndexMethod();

    //搜索
    public FullTextResult doQuery(FullTextSearchParams fullTextSearchParams);
}

2 工厂模式启动服务

对启动Lucene和Solr服务部分,使用了工厂模式 + 反射机制,根据传入的参数确定启动的服务。
代码如下

public class ServerFactory {

    /**
     * @Description: 启动服务的工厂类
     * @return:
     * @date: 2017-8-30  
     */
    public FullTextService beginService(Map<String,String> configParams){
        FullTextService fullTextService = null;
        try {
            String serverName = configParams.get("serverName");
            String type = configParams.get("type");
            String className = configParams.get("className");
            Class<?> c = Class.forName(className);
            if("solr".equals(type)){
                String url = configParams.get("url");           
                if(StringUtils.isEmpty(url)){
                    url = StringUtils.getConfigParam(ConstantParams.SOLR_URL, "", ConstantParams.SEARCH_CONFIG);
                }
                fullTextService = (FullTextService)c.newInstance();
                fullTextService.beginService(serverName,url);
            }
            if("lucene".equals(type)){
                String flag = configParams.get("flag");
                String indexPath = configParams.get("indexPath");
                fullTextService = (FullTextService)c.newInstance();
                if(StringUtils.isEmpty(indexPath)){
                    fullTextService.beginService(serverName, flag);
                }else{
                    fullTextService.beginService(serverName, flag, indexPath);
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return fullTextService;
    }
}

3 Lucene服务

针对Lucene的服务,封装的代码。下面分块讲解。
常用参数。

public class LuceneService extends FullTextServiceImpl {
        private String serverName;
    public static Map<String,IndexWriter> writerMap = new HashMap<String,IndexWriter>();
    public static Map<String,IndexSearcher> searchMap = new HashMap<String,IndexSearcher>();
    private static Analyzer analyzer = new IKAnalyzer();
    private static String indexPath = StringUtils.getConfigParam(ConstantParams.INDEXPATH, "", ConstantParams.SEARCH_CONFIG);
}

根据服务名称,索引还是搜索参数,启动相应服务。第二种方法可以传入索引地址。

@Override
    public int beginService(String serverName, String flag) {
        try {
            if("writer".equals(flag)){
                IndexWriter indexWriter = writerMap.get(serverName);
                    if(indexWriter == null){
                        Directory dir = FSDirectory.open(new File(indexPath));
                        IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_46, analyzer);
                        indexWriter = new IndexWriter(dir, config);
                        writerMap.put(serverName, indexWriter);
                    }
                    if(indexWriter != null){
                        return 1;
                    }
            }
            if("search".equals(flag)){
                IndexSearcher indexSearcher = searchMap.get(serverName);
                if(indexSearcher == null){
                    Directory dir = FSDirectory.open(new File(indexPath));
                    IndexReader reader = DirectoryReader.open(dir);
                    indexSearcher = new IndexSearcher(reader);
                    searchMap.put(serverName, indexSearcher);
                }
                if(indexSearcher != null){
                    return 1;
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return -1;
    }

@Override
    public int beginService(String serverName, String flag,String indexPath) {
        try {
            if("writer".equals(flag)){
                IndexWriter indexWriter = writerMap.get(serverName);
                if(indexWriter == null){
                    Directory dir = FSDirectory.open(new File(indexPath));
                    IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_46, analyzer);
                    indexWriter = new IndexWriter(dir, config);
                    writerMap.put(serverName, indexWriter);
                }
                if(indexWriter != null){
                    return 1;
                }
            }
            if("search".equals(flag)){
                IndexSearcher indexSearcher = searchMap.get(serverName);
                if(indexSearcher == null){
                    Directory dir = FSDirectory.open(new File(indexPath));
                    IndexReader reader = DirectoryReader.open(dir);
                    indexSearcher = new IndexSearcher(reader);
                    searchMap.put(serverName, indexSearcher);
                }
                if(indexSearcher != null){
                    return 1;
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return -1;
    }

建立索引:通过判断输入值的类型,将其加入到对应的field中,还封装了preIndexMethod();和afterIndexMethod();方法。

@Override
    public void doIndex(FullTextIndexParams fullTextIndexParams) {
        long preStart = System.currentTimeMillis();
        preIndexMethod();
        long preEnd = System.currentTimeMillis();
        System.out.println("Your preIndex spent on "+(preEnd-preStart)+" ms.");
        
        try {
            List<Map<String,Object>> listData = fullTextIndexParams.getIndexData();
            for(Map<String,Object> map : listData){
                Document doc = new Document();
                Set<String> set = map.keySet();
                Iterator<String> iter = set.iterator();
                while(iter.hasNext()){
                    String key = iter.next();
                    Object value = map.get(key);
                    IndexableField field = null;
                    if(value instanceof Integer){
                        field = new IntField(key,(Integer)value,Field.Store.YES);
                    }else if(value instanceof Long){
                        field = new  LongField(key,(Long)value,Field.Store.YES);
                    }else if(value instanceof Double){
                        field = new  DoubleField(key,(Double)value,Field.Store.YES);
                    }else if(value instanceof Float){
                        field = new  FloatField(key,(Float)value,Field.Store.YES);
                    }else{
                        field = new TextField(key,value.toString(),Field.Store.YES);
                    }
                    doc.add(field);
                }
                writerMap.get(this.serverName).addDocument(doc);
            }
            writerMap.get(this.serverName).commit();
            writerMap.get(this.serverName).close();
            writerMap.put(this.serverName, null);
        } catch (Exception e) {
            e.printStackTrace();
        }
        
        long afterStart = System.currentTimeMillis();
        afterIndexMethod();
        long afterEnd = System.currentTimeMillis();
        System.out.println("Your afterIndex spent on "+(afterEnd-afterStart)+" ms again.");
    }

搜索部分,注意其中对于搜索技术的使用。

@Override
    public FullTextResult doQuery(FullTextSearchParams fullTextSearchParams) {
        FullTextResult fullTextResult = new LuceneResult();
        try {
            String queryWord = fullTextSearchParams.getQueryWord();
            if(StringUtils.isEmpty(queryWord)){
                return null;
            }
            
            //指定搜索域
            List<String> assignmentFields = fullTextSearchParams.getAssignmentFields();
            String[] fields = null;
            if(assignmentFields != null && assignmentFields.size() > 0){
                int size = assignmentFields.size();
                fields = new String[size];
                for(int i=0;i<size;i++){
                    fields[i] = assignmentFields.get(i);
                }
            }else{
                fields = new String[]{};
            }
            
            //指定权重
            Map<String,Float> boost = fullTextSearchParams.getBoost();
            
            QueryParser queryParser = new MultiFieldQueryParser(Version.LUCENE_46,fields,analyzer,boost);
            Query query = queryParser.parse(queryWord);
            
            //返回结果数
            int returnNums = fullTextSearchParams.getReturnNums();
            
            //过滤
            Map<String,String> filterField = fullTextSearchParams.getFilterField();
            Filter filter = null;
            if(filterField != null && filterField.size() > 0){
                Set<String> set = filterField.keySet();
                Iterator<String> iter = set.iterator();
                while(iter.hasNext()){
                    String key = iter.next();
                    String value = filterField.get(key);
                    filter = new QueryWrapperFilter(new TermQuery(new Term(key,value)));
                }
            }
            
            boolean isHighlight = fullTextSearchParams.getIsHighlight();
            Highlighter highlighter = null;
            if(isHighlight){
                String preTag = fullTextSearchParams.getPreHighlight();
                String postTag = fullTextSearchParams.getPostHighlight();
                Formatter formatter = new SimpleHTMLFormatter(preTag,postTag);
                Scorer fragmentScorer = new QueryScorer(query);
                highlighter = new Highlighter(formatter, fragmentScorer);
                Fragmenter fragmenter = new SimpleFragmenter(fullTextSearchParams.getViewNums());
                highlighter.setTextFragmenter(fragmenter);
            }
            
            //显示域
            String[] viewFields = fullTextSearchParams.getViewFields();
            
            TopDocs topDocs = searchMap.get(this.serverName).search(query,filter, returnNums);
            int hits = topDocs.totalHits;
            fullTextResult.setNumFound(Long.valueOf(hits));
            ScoreDoc[] scoreDoc = topDocs.scoreDocs;
            Map map = null;
            List list = new ArrayList();
            for(ScoreDoc sd : scoreDoc){
                map = new HashMap();
                int docID = sd.doc;
                Document doc = searchMap.get(this.serverName).doc(docID);
                
                String highlightResult = null;
                if(viewFields != null && viewFields.length > 0){
                    for(String vf : viewFields){
                        if(highlighter != null){
                            highlightResult = highlighter.getBestFragment(analyzer, vf,doc.get(vf));
                        }
                        if(highlightResult != null){
                            map.put(vf, highlightResult);
                        }else{
                            String value = doc.get(vf);
                            int docLength = value.length();
                            if(docLength!=0){
                                if(docLength > fullTextSearchParams.getViewNums()){
                                    value = value.substring(0,fullTextSearchParams.getViewNums());
                                }
                                map.put(vf, value);
                            }
                            else{
                                return null;
                            }
                        }
                    }
                    list.add(map);
                }else{
                    for(IndexableField field : doc.getFields()){
                        if(highlighter != null){
                            highlightResult = highlighter.getBestFragment(analyzer, field.name(),field.stringValue());
                        }
                        if(highlightResult != null){
                            map.put(field.name(), highlightResult);
                        }else{
                            String value = field.stringValue();
                            int docLength = value.length();
                            if(docLength > fullTextSearchParams.getViewNums()){
                                value = value.substring(0,fullTextSearchParams.getViewNums());
                            }
                            map.put(field.name(), value);
                        }
                    }
                    list.add(map);
                }
            }
            fullTextResult.setResultList(list);
            searchMap.put(this.serverName, null);//关掉之后会重新启动,达到实时
        } catch (Exception e) {
            e.printStackTrace();
        }
        return fullTextResult;
    }

删除索引

@Override
    public void deleteIndex(FullTextIndexParams fullTextIndexParams) {
        preDeleteIndexMethod();
        try {
            String id = fullTextIndexParams.getId();
            writerMap.get(this.serverName).deleteDocuments(new Term("docfullid",id));
            writerMap.get(this.serverName).commit();
            writerMap.get(this.serverName).close();
            writerMap.put(this.serverName, null);//关掉之后会重新启动,达到实时
        } catch (Exception e) {
            e.printStackTrace();
        }
        
        afterDeleteIndexMethod();
    }

4 Solr服务

需要参数

public Map<String,SolrServer> solrServerMap = new HashMap<String,SolrServer>();
private String serverName;

启动服务,只需要知道服务URL。

@Override
    public int beginService(String serverName) {
        SolrServer solrServer = solrServerMap.get(serverName);
        if(solrServer == null){
            solrServer = beginServer();
            solrServerMap.put(serverName,solrServer);
            return 1;
        }
        return -1;
    }

    @Override
    public int beginService(String serverName, String url) {
        if(StringUtils.isEmpty(url)){
            return -1;
        }
        SolrServer solrServer = solrServerMap.get(serverName);
        if(solrServer == null){
            solrServer = beginServer(url);//启动solr服务
            solrServerMap.put(serverName,solrServer);//存储启动的服务
            return 1;
        }
        return -1;
    }

    public SolrServer beginServer(){
        SolrServer solrServer = null;
        try {
            String url = StringUtils.getConfigParam(ConstantParams.SOLR_URL, "", ConstantParams.SEARCH_CONFIG);
            solrServer = new HttpSolrServer(url);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return solrServer;
    }

    public SolrServer beginServer(String url){
        SolrServer solrServer = null;
        try {
            solrServer = new HttpSolrServer(url);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return solrServer;
    }

建索引,因为type、fields和其他的一些缺省设置,在schema.xml已经配置好了,所以只需要将参数加入SolrInputDocument就好。

    @Override
    public void doIndex(FullTextIndexParams fullTextIndexParams) {
        try {
            List<Map<String,Object>> indexData = fullTextIndexParams.getIndexData();
            if(indexData != null && indexData.size() > 0){
                List<SolrInputDocument> documentList = new ArrayList<SolrInputDocument>();
                SolrInputDocument doc = null;
                //将Map中的内容取出,加入到doc,docList,然后建立索引
                for(Map<String,Object> map : indexData){
                    doc = new SolrInputDocument();
                    Set<String> set = map.keySet();
                    Iterator<String> iter = set.iterator();
                    while(iter.hasNext()){
                        String key = iter.next();
                        Object value = map.get(key);
                        doc.addField(key, value);
                    }
                    documentList.add(doc);
                }
                System.out.println("======================"+this.solrServerMap.get(this.serverName));
                this.solrServerMap.get(this.serverName).add(documentList);
                this.solrServerMap.get(this.serverName).commit();
            }else{
                return;
            }
        } catch (SolrServerException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
        long afterStart = System.currentTimeMillis();
        afterIndexMethod();
        long afterEnd = System.currentTimeMillis();
        System.out.println("Your afterIndex spent on "+(afterEnd-afterStart)+" ms again.");
    }

搜索部分

@Override
    public FullTextResult doQuery(FullTextSearchParams fullTextSearchParams) {
        FullTextResult result = new SolrResult();
        try {
            String queryWord = fullTextSearchParams.getQueryWord();
            if(StringUtils.isEmpty(queryWord)){
                return null;
            }
            List<String> assignmentFields = fullTextSearchParams.getAssignmentFields();
            List<Map<String,String>> assignFields = fullTextSearchParams.getAssignFields();
            String queryString = "";
            if(assignmentFields != null && assignmentFields.size()>0){
                for(String assignmentField : assignmentFields){
                    queryString += assignmentField+":"+queryWord+" OR ";
                }
                int pos = queryString.lastIndexOf(" OR ");
                queryString = queryString.substring(0, pos);
            }else if(assignFields != null && assignFields.size()>0){
                //title:中国
                String lastValue = "";
                for(Map<String,String> assignField : assignFields){
                    Set<String> set = assignField.keySet();
                    Iterator<String> iter = set.iterator();
                    while(iter.hasNext()){
                        String key = iter.next();
                        String value = assignField.get(key);
                        queryString += key+":"+queryWord + ConstantParams.SINGLE_BLANK + value + ConstantParams.SINGLE_BLANK;
                        lastValue = value;
                    }
                }
                int pos = queryString.lastIndexOf(" "+lastValue+" ");
                queryString = queryString.substring(0, pos);
            }else{
                queryString = queryWord;
            }

            System.out.println("queryString:"+queryString);
            SolrQuery params = new SolrQuery(queryString);

            //设置显示域
            String[] viewFields = fullTextSearchParams.getViewFields();
            params.setFields(viewFields);

            //高亮参数
            boolean isHighlight = fullTextSearchParams.getIsHighlight();
            String[] highlightFields = fullTextSearchParams.getHighlightFields();
            if(isHighlight && highlightFields != null && highlightFields.length > 0){
                params.setHighlight(true);
                params.setHighlightSimplePre(fullTextSearchParams.getPreHighlight());
                params.setHighlightSimplePost(fullTextSearchParams.getPostHighlight());
                params.setHighlightFragsize(fullTextSearchParams.getViewNums());
            }
            //          Fragmenter fragmenter = new SimpleFragmenter(fullTextSearchParams.getViewNums());

            //排序域 String:需要排序的域名,Boolean:true 升序 false 降序
            Map<String,Boolean> sortField = fullTextSearchParams.getSortField();
            if(sortField != null){
                Set<String> set = sortField.keySet();
                Iterator<String> iter = set.iterator();
                while(iter.hasNext()){
                    String key = iter.next();
                    Boolean value = sortField.get(key);
                    if(value){
                        params.addSort(key,ORDER.asc);
                    }else{
                        params.addSort(key,ORDER.desc);
                    }

                }
            }
            //过滤域
            Map<String,String> filterField = fullTextSearchParams.getFilterField();
            if(filterField != null && filterField.size() > 0){
                StringBuilder str = new StringBuilder();
                Set<String> set = filterField.keySet();
                Iterator<String> iter = set.iterator();
                while(iter.hasNext()){
                    String key = iter.next();
                    String value = filterField.get(key);
                    str.append(key+":"+value);
                    str.append("-vertical-");
                }
                String[] fieldFields = str.toString().split("-vertical-");
                params.addFilterQuery(fieldFields);
            }

            //开始行
            params.setStart(fullTextSearchParams.getStartNums());
            //返回的总结果数
            params.setRows(fullTextSearchParams.getReturnNums());
            //统计域
            boolean isFacet = fullTextSearchParams.getIsFacet();
            String[] facetFields = fullTextSearchParams.getFacetFields();
            if(isFacet && facetFields != null && facetFields.length>0){
                params.addFacetField(facetFields);
            }

            QueryResponse response = this.solrServerMap.get(this.serverName).query(params);
            SolrDocumentList list = response.getResults();
            result.setNumFound(list.getNumFound());

            SolrDocument document = new SolrDocument();
            SolrDocumentList hlList = new SolrDocumentList();

            //高亮结果
            if(isHighlight && highlightFields != null && highlightFields.length > 0){
                Map<String,Map<String,List<String>>> map = response.getHighlighting();
                for(int i=0;i<list.size();i++){
                    for(int j=0;j<highlightFields.length;j++){
                        document = list.get(i);
                        if(map != null && map.get(document.getFieldValue("docfullid")) != null && map.get(document.getFieldValue("docfullid")).get(highlightFields[j]) != null){
                            if("datetime".equals(map.get(document.getFieldValue("docfullid")).get(highlightFields[j]))){
                                document.setField(highlightFields[j], map.get(document.getFieldValue("docfullid")).get(highlightFields[j]).get(0)); 
                            }else{
                                document.setField(highlightFields[j], map.get(document.getFieldValue("docfullid")).get(highlightFields[j]).get(0)); 
                            }
                        }else{
                            //无高亮信息,控制字数
                            String temp = (String)document.getFieldValue(highlightFields[j]);
                            if(temp.length()>fullTextSearchParams.getViewNums()){
                                String string = temp.substring(0,fullTextSearchParams.getViewNums());
                                document.setField(highlightFields[j], string);
                            }
                        }
                    }
                    hlList.add(document);
                }
                result.setResultList(hlList);
            }else{  
                result.setResultList(list);
            }

            //统计结果
            if(isFacet && facetFields != null && facetFields.length>0){
                List<FacetField> listField = response.getFacetFields();
                result.setFacetList(listField);
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
        return result;
    }

代码和架构都很好,希望能时常看看。

END

相关文章

  • Lucene和Solr服务封装

    1 架构设计2 工厂模式启动服务3 Lucene服务4 Solr服务 这个是将Lucene和Solr的服务进行封装...

  • ElasticSearch

    ElasticSearch是一个基于Lucene的搜索服务器,类似于Solr实现了对Lucene的封装。它提供了一...

  • solr

    何为solr solr是java搜索引擎Lucene的更高一层封装,通过webapp服务器实现可视化界面,方便使用...

  • Solr安装配置简介

    什么是Solr Solr是一个高性能,采用Java5开发,基于Lucene的全文搜索服务器。Solr对Lucene...

  • Solr 原理篇

    Solr和ElasticSearch一样,都是基于Lucene的搜索服务器。 Solr 在传统的搜索应用中表现好于...

  • solr6.5搜索引擎技术

    搜索技术 SOLR和LUCENE关系 Solr是基于Lucene做的,Lucene是一套信息检索工具包,但并不包...

  • Solr

    一、Solr简介 1、Solr是什么 Solr是一个基于Lucene的Java搜索引擎服务器。Solr 提供了层面...

  • Lucene编译

    编译 直接在lucene-solr目录下执行:ant compile 到lucene-solr/lucene目录下...

  • 全文搜索引擎Solr原理和实战教程

    Solr简介 1.Solr是什么? Solr它是一种开放源码的、基于 Lucene Java 的搜索服务器,易于加...

  • solr4.10.3集成tomcat

    从lucene的官方下载,lucene和solr的版本是同步更新的:http://lucene.apache.or...

网友评论

      本文标题:Lucene和Solr服务封装

      本文链接:https://www.haomeiwen.com/subject/tkpniftx.html